Three days later, the British retreated, and when the Madisons returned to the White House, they found an empty shell. The president and his wife moved to Octagon House in Washington, while the White House was rebuilt. The war effort began improving. The British were first turned back in upper New York and then at Baltimore.